Image Similarity Based on Direct Human Judgment

Recently the field of human-based computation has proposed new ways to label and locate objects in images by using games or websites like Mechanical Turk. This paper proposes a way to use human judgements, gathered through Mechanical Turk, to measure image similarity. We then compare the proposed method to other popular image similarity methods and metrics. The results show that collective human judgment on image similarity is highly variable because it depends on the context of the humans submitting their judgement. This makes collective human judgement for image similarity more powerful than the current popular methods, but also harder to harness.